Music video game with user directed sound generation

ABSTRACT

A music based video game provides a user directed sound generation feature. In some embodiments, the user is provided a video game controller simulating a musical instrument, which the user may operate similarly to its real musical instrument counterpart. A free play feature is provided for users to operate the video game controller, where a processor of a video game console receives input signals from the video game controller and outputs audio samples based on the received input signals. During the free play feature, multiple suggestions as to how to operate the video game controller are simultaneously provided to give users some guidance as to what to play using the video game controller.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a divisional of U.S. patent application Ser.No. 14/733,541, filed Jun. 8, 2015, which is a divisional of U.S. patentapplication Ser. No. 12/172,927, filed Jul. 14, 2008, now U.S. Pat. No.9,061,205, the disclosures of which are incorporated by referenceherein.

BACKGROUND

The present invention relates generally to video games, and moreparticularly to a music based video game with a guitar shaped gamecontroller.

Video games provide a source of entertainment for many. Different videogames provide users with different interactive challenges, many of whichsimulate interesting situations and scenarios that users may nottypically experience. Often, video game simulations are simplified butdifferent versions of their real counterparts, and proficiency at thevideo games require practice using different types of skill sets. Videogames may also provide venues for users to express their creativity in avariety of different ways.

Music based video games with simulated musical instruments are popularamongst some users. Such music based video games may allow users to playalong with songs, for example by providing a predetermined sound filefor use in providing audio responsive to video game play. Video gamecontrollers used in conjunction with the music based video games may bedesigned in the general shape of different musical instruments, whereoperation of the video game controllers may simulate the operation ofthe musical instruments they resemble. Use of a video game controller inthe general shape of a musical instrument may provide a user a morerealistic experience as a musician. Video game controllers are also moreversatile than real instruments, and may be utilized to generate a muchwider range of audio samples than a conventional instrument.

Merely playing along with a song, for example that offers only apredetermined musical output however, may not fully simulate the musicalexperience, particularly the creative aspects of music creation.Experienced musicians may improvise notes, beats, vocals, chords, riffsand solos when playing music in a group setting, and inexperiencedmusicians or others may not comprehend or have the capability to do soin a harmonious manner. Many musicians do not merely play songs createdby others, but also compose and create their own songs, as well asembellish the works of others. Creating a song, or any piece of music isoften difficult, particularly for someone who may not be fully versed inmusic theory or application.

SUMMARY OF THE INVENTION

The invention provides a music based video game with user directed soundgeneration and, in some embodiments, local and remote group musicalcollaboration. In one aspect, the invention provides a method forgenerating audio samples within a song or style of musical accompanimentin a music based video game, comprising: determining an angular positionof a video game controller; determining a status of a plurality of inputdevices of the video game controller; selecting an audio sample based onthe angular position or other position and the status; and commanding anoutput of the selected audio sample.

In another aspect, the invention provides a method of complying with oneof a plurality of suggested note sequences in a musical video game,comprising: presenting a plurality of sequences of suggested userinputs; receiving an input signal generated based on user operation ofinput devices on a video game controller, the input signal in compliancewith a first suggested input of at least one of the plurality ofsequences; determining compliance of subsequently received input signalsfrom the video game controller with the at least one of the plurality ofsequences; and replacing the at least one of the plurality of sequenceswith a new sequence of suggested user inputs.

In another aspect, the invention provides a system for providing avisual interface for recommending note, chord, beat or vocal sequencesin a musical video game, comprising: a display; a guitar shaped videogame controller including a body, a neck extending from the body, astrum input on the body, and a plurality of fret inputs on the neck;memory, the memory including display information of a plurality ofcards, each card including suggested sequences of fret inputcombinations executable on the plurality of fret inputs on the videogame controller; and a processor configured by program instructions togenerate display information, the program instructions includinginstructions for: displaying at least two of the plurality of cards;receiving input signals generated by user operation of the video gamecontroller, the input signals based on fret input combinations executedon the video game controller; determining successful compliance of theinput signals with one card of the at least two cards; replacing the onecard with another card of the plurality of cards.

In another aspect, the invention provides a method of generating a musiccompilation, comprising: receiving an indication of the video gamecontroller positioned in a first of a plurality of angular regions, theangular regions based on physical orientation of the video gamecontroller; receiving a first input signal from the video gamecontroller; outputting an audio track based on the first input signal;receiving an indication of the video game controller entering a secondof the plurality of angular regions; receiving a second input signalfrom the video game controller; and outputting an audio sample based onthe second input signal, the audio sample outputted simultaneously withthe audio track.

In another aspect, the invention provides a method of transitioning froma first section of a song to a second section of the song in a musicbased video game, comprising: providing a first plurality of suggestedinput sequences, the input sequences including suggested user operationof a video game controller, the first plurality of suggested inputsequences suitable for the first section of the song; providing atransition input sequence; receiving input signals from the video gamecontroller, the input signals compliant with the transition inputsequence; replacing the first plurality of suggested input sequenceswith a second plurality of suggested input sequences, the secondplurality of suggested input sequences suitable for the second sectionof the song.

In yet another aspect, the invention provides a method of generatingaudio tracks in a music video game, comprising: receiving a first inputsignal indicating operation of a first plurality of input devices;outputting a polyphonic audio compilation based on the first inputsignal, the polyphonic audio compilation including at least two audiotracks featuring different instruments; receiving a second input signalindicating operation of a second plurality of input devices; selectingat least one of the at least two audio tracks based on the second inputsignal; and muting the at least one of the at least two audio tracks.

These and other aspects of the invention are more fully comprehendedupon review of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example of a video game system in accordance with aspectsof the invention;

FIG. 2 is an example of a block diagram of a video game consoleprocessing unit in accordance with aspects of the invention;

FIG. 3 is a flow diagram of a process of determining an audio output inaccordance with aspects of the invention;

FIG. 4 is a flow diagram of the process of adjusting a mode of a videogame controller based on an angular position of the video gamecontroller;

FIG. 5 is an illustration of a guitar shaped video game controller witha different mode assigned to each of three predefined tilt regions inaccordance with aspects of the invention;

FIG. 6 is a flow diagram of a process of presenting note suggestioncards to users in accordance with aspects of the invention;

FIG. 7 is a screen shot of a user interface of a music based video gameduring user directed sound generation in accordance with aspects of theinvention;

FIG. 8 is a flow diagram of a process of replacing cards which have beencompleted during video game play in accordance with aspects of theinvention;

FIG. 9 is a flow diagram of a process of transitioning from a firstsection of a song to a second section of a song in accordance withaspects of the invention;

FIG. 10 is a flow diagram of an alternate process of determining audiooutputs in accordance with aspects of the invention;

FIG. 11 is a flow diagram of a dynamic process of determining audiooutputs in accordance with aspects of the invention; and

FIG. 12 is a flow diagram of a process of generating and recording amulti-dimensional or polyphonic audio compilation in accordance withaspects of the invention.

DETAILED DESCRIPTION

FIG. 1 is an example of a video game system in accordance with aspectsof the invention. The video game system includes a video game console101, a display 103, and a guitar shaped video game controller 105. Thevideo game console includes internal circuitry which allows the consoleto run a video game by executing various program instructions related toproper execution of the video game. The video game console typicallyincludes one or more processors, memory, and various interface circuitryfor running a video game. The instructions for each specific video gameare generally found on a removable memory source such as a video gameCD-ROM, inserted into a removable memory interface 111 of the video gameconsole. In some embodiments of the invention, the video game consolemay alternatively be a handheld gaming device, including similarinternal circuitry as herein described, as well as, for example, a builtin display or displays and various different user input devices.

The video game console may communicate with one or more video gamecontrollers to properly run the video game. The video game controller inthe embodiment of FIG. 1 is a guitar shaped video game controller, andincludes a guitar body 113 with a neck 115 extending from the body. Theguitar neck includes a plurality of fret buttons 117 and an analogslider input 119, while the body includes a strum bar 121, a tremolo arm123, and a palm mute button 125. In various embodiments additional inputdevices may also be provided. The guitar shaped video game controllermay be used, for example, to play a rhythm based music video game. Inother embodiments of the invention, video game controllers withdifferent configurations may be utilized depending on the format of eachindividual video game. For example, an embodiment of the invention mayinclude a video game controller simulating a drum set, with the videogame console configured to generate drum audio samples, or othersamples, and display a user interface with more drum relatedinformation. Likewise, an embodiment of the invention may include amicrophone, with tilt or position sensing in some embodiments, with thevideo game console configured to process inputs from the microphone, andprovide vocal effects or generate instrument sounds based on the pitchor rhythm of the user's voice and in some embodiments position ororientation of the microphone.

Generally, a user of the video game system operates the video gamecontroller to generate user input signals, which are sent to the videogame console. In music video games using video game controllersfashioned after musical instruments, for example, the guitar shapedvideo game controller of FIG. 1, operation of the video game controllermay mimic or simulate the operation of the video game controller's realinstrument counterpart. For the aforementioned guitar shaped video gamecontroller, a user may simulate different guitar chord fingeringsthrough operation of the fret buttons, for example, different fretfingering combinations, and may simulate a strum of a guitar through anupward or downward deflection of the strum bar. Control circuitry withinthe guitar shaped video game controller may generate input signals basedon the user operation of these input devices on the video gamecontroller as well as the operation of various other different inputdevices on the video game controller.

Input signals generated by the video game controller may includeinformation on fret combinations a user performs on the video gamecontroller when the strum input is actuated. In some embodiments of theinvention, alternative ways of generating input signal combinationsthrough the video game controller are also possible, for example,touching the analog slide input without a strum actuation, or forexample, knocking on the body of the video game controller with theuser's hand. Often, the video game controller is operated in response tovideo game instructions provided by the video game console to the user.Input signals generated by the video game controller are processed bythe video game console to personalize video game play based on theuser's unique operation of the video game controller. In the embodimentof FIG. 1, the input signals are sent to the video game console througha wireless connection 127, but in some embodiments, the video gameconsole and video game controller may be connected via a wiredconnection. In embodiments of the invention, the input signals aregenerally processed by the video game console to generate audio outputsassociated with video game play. For example, the console may use inputsignals received from a guitar shaped video game controller to generatevarious guitar based audio samples.

In some embodiments, the available inputs on a guitar shaped video gamecontroller may also be used to generate other possible sounds in themusic video game. For example, in some embodiments, different inputsignals from the guitar shaped video game controller may be processedand mapped by the video game console into drum output sounds, drumbeats, vocal outputs, DJ or keyboard output sounds. In this fashion,users of the video game may generate sounds based on various differentinstruments using the guitar shaped video game controller.

The video game console is also in communication with a display unit,generally through an audio-video cable 129 or similar wired connection.The display unit is typically a television, with a display screen 131and at least one audio output device, such as a speaker 133. In theembodiment of FIG. 1, the display screen shows a screen shot of a videogame play interface in a music video game. The game play interface ispresented while the video game is performing user directed soundgeneration, that is, a free play option which allows users of the musicvideo game to simulate music creation through operation of the guitarshaped video game controller in place of a real guitar. Some embodimentsof the invention include a video game controller with an accelerometer,an infrared sensor, magnetic position or similar position or motionsensors, to determine a vertical tilt level of the video gamecontroller. The game play interface includes a tilt indicator 135indicating a current tilt level 137 of the guitar shaped video gamecontroller, and one of a number of tilt regions 139 in which the videogame controller is currently located. In some embodiments of theinvention, the tilt region in which the video game controller ispositioned affects the processing by the video game console of inputsignals received from the video game controller. The game play interfacealso includes multiple cards or visual displays 141 displaying differentnote sequence suggestions for players to follow in case the users desirerecommendations as to how to operate the video game controller. The notesequence suggestions include a number of different recommended fretinput sequences the user may perform on the video game controller.However, in embodiments of the invention, game play and advancing in thevideo game are not predicated on compliance with any of the presentednote sequences. The video game console may process the input sequencesinto audio outputs which fit into the context of a current audio trackbeing played in the background of the video game. Various alternativenote recommendations or instructions may also be incorporated into thevideo game, such as for example, different sequences of visualized notesarranged onto multiple music tracks. The game play interface alsoincludes a scrolling music track 143, which in some embodiments, maydisplay recent fret, vocal, keyboard, drum or other instrument inputcombinations, for example, combination 145, performed on the video gamecontroller by the user. The fret combinations may be visualized by, forexample, gems representative of recently activated inputs or some subsetof those inputs. Furthermore, the scrolling music track may also includerelevant tilt information, for example, different shaped gems or gemsraised different distances from a surface of the music track, toindicate a particular angular configuration at which each fretcombination was performed. Embodiments of the invention including a tiltfeature may also include note recommendations including suggestedoperation of the tilt feature. In some embodiments information of uservideo game play for a sequence of cards is retained, and then displayedon a note highway as user instructions for another user. If, forexample, the other user successfully complies with the userinstructions, the other user may engage in video game play for a furthersequence of cards, with information of the other user game play for thefurther sequence of cards retained, and then displayed on the notehighway as user instructions for the first user. Extent user compliancemay be used to provide a score for each user, and, in some embodiments,such game play may continue until a user is unable to comply with a setof user instructions.

FIG. 2 is an example of a block diagram of a video game consoleprocessing unit in accordance with aspects of the invention. In theembodiment of FIG. 2, the processing unit includes at least oneprocessor 201 interconnected with other components via a system bus 203.The other components may include, for example, a main memory 205 of thevideo game console, a removable memory interface 207, a userinput/output port 209, a wireless transceiver 211, an audio driver 213,a video driver 215, an Internet input/output port 217, and othercircuitry 219, which may include for example an infrared sensor. Inother embodiments of the invention, there may be different combinationsof components that make up a video game console processing unit,depending on the individual needs of each application.

The processor executes various types of software instructions tofacilitate video game play of, for example, the music based video gamedescribed with respect to FIG. 1. The processor may use the componentsof the processing unit in order to execute the software instructions.The processor may retrieve video game instructions for a specific videogame from a removable memory source, for example, a video game CD-ROM,inserted into the removable memory interface of the video game console.The processor may process the video game instructions in accordance withconsole specific program instructions, which are generally found in themain memory of the video game console. The processor also receives userinput signals from a connected video game controller, either through theuser input/output port or the wireless transceiver. The processorprocesses the various instructions and received input signals togenerate audio and video output signals representative of video gameplay.

The processor may also be in data communication with a display unit,which outputs audio and video outputs of video game action to a user ofthe system. The processor may send audio generation information to theaudio driver, and video generation information to the video driver, eachof which generates audio and video output signals, respectively, fromthe received generation information. The audio and video drivers forwardthe audio and video output signals through a combined displayinput/output port 221, or alternatively, separate audio and videoinput/output ports, to the display unit.

In some embodiments, the processor is also connected to the Internet viaeither the Internet input/output port, or via the wireless transceiver.A connection to the Internet may be used to facilitate multiplayer gameplay with other users in remote locations if a multiplayer option isprovided by a particular video game. In some embodiments, an Internetconnection may also be used by video game companies to offer, for freeor for sale, various downloadable content associated with a particulargame. In the music based video game of FIG. 1, downloadable content mayinclude, for example, different characters, lessons, otherwiseunavailable note suggestion cards, and new songs or background audiotracks. In other embodiments, the invention may provide for creation ofnote suggestion cards, and the Internet connection may facilitatesharing or trading of user created cards between users in remotelocations.

FIG. 3 is a flow diagram of a process of determining an audio output inaccordance with aspects of the invention. In most embodiments theprocess is performed by a video game console, such as the video gameconsole of FIG. 1, on a processor, for example the processor of FIG. 2.In some embodiments a video game controller, for example, the guitarshaped video game controller described with respect to FIG. 1, generatesinput signals based on the manner in which the video game controller isoperated. A processor of the video game console receives the inputsignals from the video game controller, and processes the input signalsto generate audio, and in some embodiments video, outputs. In musicalvideo games, for example as have been described, the audio outputs maybe in the form of audio samples recorded from real guitars, otherinstruments, for example, bass guitars, drum sets, or pianos orkeyboards, or other sound generators, with or without effectsprocessing. In some embodiments, the audio samples may be, for example,synthesized audio samples under the Musical Instrument DigitalInterface, or MIDI, protocol. In these embodiments, audio samples forany of a wide range of instruments may be assigned to different inputsignals generable from the video game controller. Therefore, a guitarshaped video game controller, such as the one described with respect toFIG. 1, may be operated to generate not only guitar audio samples, butmay be easily configured to generate audio samples from drum sets andother types of instruments through the video game console as well. Inthe embodiment of FIG. 3, the video game controller includes at least astrum input, a plurality of fret inputs, and a position detectiondevice, for example, an accelerometer.

In block 311, the process receives a strum input signal from the guitarshaped video game controller. A strum input signal may be generated whena simulated strum is performed on the video game controller. A strum maybe performed by, for example an upward or downward deflection of thestrum input device, simulating a strumming of the strings on a realguitar. In some embodiments, strum input signals may also include otherinformation, for example, information of the direction of the strum, theaverage speed of a series of consecutive strums, or for example,information of the strength of a strumming action or other actuations ofinputs. A processor receives the strum input signal and may determine,for example, the timing of an audio output based on the timinginformation of the strum. In embodiments where the strum input signalincludes additional information, the processor of the video game consolemay adjust certain audio output properties, for example, volume ormagnitude of output, based in part on the additional informationincluded in the strum input signal.

In block 313, the process determines a status of each of the pluralityof fret inputs of the video game controller at the time the strum inputwas activated. The fret inputs are located on the neck of the video gamecontroller, and operation of the fret inputs simulates chord fingeringsperformed on the strings of a real guitar. In an exemplary embodiment ofthe invention, the guitar shaped video game controller provides fivefret input buttons. Additionally, five touch sensitive buttons may beprovided in some embodiments. Generally, video game instructionsdetermine the mapping of audio samples to particular combinations ofactivated fret input buttons, much like different string fingerings andmotions may produce different notes on a real guitar. In someembodiments, users may also adjust configuration settings associatedwith the video game to change the associations between audio samples andfret combinations.

Some embodiments of the invention may also include additional inputdevices on the video game controller, for example, the analog touch padinput or the palm mute button described with respect to the video gamecontroller of FIG. 1, or peripheral devices, for example, a foot pedalconnected to the video game controller. Some embodiments may alsodetermine a status of each of these additional input devices. Video gameinstructions may include information on mapping various audio samples toinput signals generable by the additional input devices. In someembodiments, video game instructions may alternatively includeinformation on alternate audio sample mappings for different executedfret combinations when one or more of the additional input devices issimultaneously depressed or activated.

In block 315, the process determines a tilt region of the video gamecontroller at the time the strum input was activated. In embodiments ofthe invention, the video game controller includes an accelerometer orsimilar position or motion detection device, such as an infrared orsimilar sensor. The accelerometer or other motion detection deviceprovides information on an orientation, which may be considered angularconfiguration or positioning of the video game controller, for example,a vertical angle at which the video game controller is tilted or held.Angular or tilt position of the video game controller may be determinedbased on the angle an axis of extension of the neck makes with ahorizontal reference, for example, the ground. A plurality of angulartilt regions are predefined by the video game. An exemplary embodimentof the invention includes three tilt regions, but other embodiments ofthe invention may include more or less than three tilt regions. A tiltregion is determined to be active if the angular position of the videogame controller falls within the tilt region. In some embodiments, thetilt region is determined by a control unit in the video gamecontroller, and an input signal sent to the video game console mayinclude information of the active tilt region. In other embodiments,input signals may include information on angular or tilt position, andthe processor of the video game console may use the input signal todetermine the active tilt region. Depending on the application, someembodiments may employ the accelerometer in a variety of different ways,for example, determining horizontal positioning of a video gamecontroller, or for example, determining a speed at which the video gamecontroller shifts from a first position to a second position.

In block 317, the process selects a note audio sample, a chord audiosample, or a sequence of audio samples to output. In some embodiments ofthe invention, the tilt region determines the type of audio outputsselected. For example, in embodiments with three tilt regions, theprocessor of the video game console may determine whether to outputeither notes, chords, or sequences of notes and/or chords based on thetilt region active at the time of strum or other user operation of avideo game controller. The different fret combinations in a first tiltregion may be designated for single notes, the fret combinations in asecond tilt region to chords, and the fret combinations in a third tiltregion to note or chord sequences. The processor of the video gameconsole selects an audio output based on information collected from thestrum input signal, the fret input signals, and the tilt region inputsignal. In this fashion, each fret combination may be processed by thevideo game console to output one of three different audio samples, basedon the active tilt region.

In block 319, the process outputs the selected audio sample or samples,whether it is a note, a chord, or an audio sample sequence. A processorof the video game console may send audio generation information to forexample, an audio driver, which in turn may generate the selected audiosample or samples to be output by a speaker or similar audio outputdevice. After the selected audio sample or samples are output, theprocess returns.

In some embodiments, particular notes and chords may be generatedindependent of the process of FIG. 3. For example, in some embodiments,in addition to the fret buttons, an analog input device may also beavailable on the neck of the video game controller. In theseembodiments, the processor may generate outputs based solely on inputsignals generated by the analog input, independent of a strum inputsignal. Therefore, some outputs or modified audio samples may begenerated when the processor receives input signals indicating onlytapping, holding or sliding a finger or similar actuator along theanalog input. In some embodiments, an accelerometer of the video gamecontroller may also be configured to detect vibrations or quick joltsapplied to the video game controller. In these embodiments, an inputsignal may be generated by the video game controller when, for example,the user knocks on the body of the video game controller with his hand.The processor of the video game controller may process this type ofinput signal, and generate an output audio sample of a percussive knockor pyrotechnic sound in response. Other embodiments of the invention mayinclude various different types of input-output mappings which eitherexpand on or deviate from the process presented in FIG. 3. In otherembodiments, one or more additional input buttons or devices may befound on the video game controller, for example, a tremolo arm, or anauxiliary or palm mute button. The additional input devices may be usedto generate input signals which the video game console processes to, forexample, determine alternative audio outputs.

FIG. 4 is a flow diagram of the process of adjusting a mode of a videogame controller based on an angular position of the video gamecontroller. In many embodiments the process is performed by a video gameconsole or a processor. As was described with respect to block 317 ofFIG. 3 above, in some embodiments video game controller includes aposition detection device which determines the angular position orconfiguration of the video game controller. The angular position isprocessed by the video game console in determining an audio sample tooutput.

In block 411, the process determines a tilt region. A processor of thevideo game console receives an input signal from the video gamecontroller including information on the angular position of the videogame controller. The angular position falls within one of a plurality ofpredefined angular tilt regions, each of which includes an upper angularlimit and a lower angular limit. The processor may determine which tiltregion the video game controller currently falls within, and determinethat region to be the active tilt region. In some embodiments, vibrationfeedback, using for example vibration generating devices, may beutilized to indicate approach to different tilt regions and/or changesin active tilt regions.

In block 413, the process determines a mode with which the active tiltregion is associated. The mode dictates how input signals generated bythe video game controller are processed by the processor of the videogame console. In some embodiments, each tilt region defines a mode, witheach mode designated by the processor for generating a particular typeof output signal. Some embodiments of the invention may include, forexample, a regular note mode, a power chord mode, and a riff mode. Someembodiments of the invention may include a regular note mode with notesin a first octave, a regular note mode with notes in a second octave,with for example the second octave lower than the first octave, and ariff mode. When a tilt region associated with the regular note mode isactive, the various fret combinations executable on the video gamecontroller may be mapped to audio samples of single notes. This may bedone, for example, by associating various fret combinations with alookup table referencing single note audio samples. Likewise, when thepower chord mode is active, each fret combination may be mapped to powerchord audio samples, and when the riff mode is active, the fretcombinations may be mapped to different riffs or similar note sequences.Riffs are rhythmic note sequences of varying length which may be used asbackground audio tracks while other notes or audio samples are beingplayed. Other similar note sequences available in a mode designated fornote sequence mappings may be, for example, solo licks, or moreintricate note sequences which may be difficult to execute or play on anote-by-note basis.

In block 415, the process maps or associates audio samples to differentfret combinations based on the determined mode. In some embodiments, avideo game CD-ROM or similar removable memory source holds informationon audio sample mappings for each mode of the video game, in the formof, for example, a lookup table. The audio samples available for thedetermined mode are mapped to different fret combinations executable onthe video game controller. Input signals received by the processor ofthe video game console indicating a particular fret combination areprocessed by the processor to generate the audio sample or sequence ofaudio samples mapped to the fret combination. In some embodiments, themappings may only apply to certain fret combinations, for example, zerofret, one fret, or two fret combinations, which are relatively easier toexecute on the video game controller. In these embodiments, the moredifficult fret combinations may be mapped to special audio samplesuseable in every mode, may be user configurable, or may not be mapped toaudio samples at all. In other embodiments, the mappings may apply toall possible fret combinations. In embodiments where each mode isdesignated for a certain type of audio output, each of the mapped audiosamples in a particular mode will be of that type. For example, powerchord mode may map all power chords to fret combinations, note mode maymap all notes, and riff mode may map riffs and similar sequences. Someembodiments may provide for note, chord, or riff configuration options.For example, users may adjust the mappings of different notes, chords,or riffs to different fret combinations, and in some embodiments thecard or scrolling notes displays as well. Some embodiments may alsoprovide for audio sample generation and recordation, and in someembodiments sharing of recorded information with remote and local users.For example, users may be allowed to create and record original riffs,and map the riffs to a fret combination.

Some embodiments of the invention may also provide users with one ormore instructions or suggestions as to how to operate the video gamecontroller for the video game console to generate pleasing or melodicsounds. As such, the suggestions may be mode-specific, and so in theseembodiments, the fret combination mapping process may be accompanied bya selection of video game controller operation instructions orsuggestions communicated to the user as well.

In block 417, the process processes video game play. Video game play mayinclude, for example, a free play feature where a user may create hisown music using a guitar shaped video game controller, much like pickingup and playing a real guitar. The user may use the fret inputs toperform different fret combinations and deflect the strum input togenerate input signals. The video game controller may receive andprocess the input signals to generate output audio, either in the formof single notes or chords, or in a string or sequence of audio samples.In the aforementioned regular note and power chord modes, each strum andaccompanying fret combination performed on the video game controller isprocessed by the video game console to generate notes and chords mappedto each fret combination. Typically audio notes and chords are outputtedat substantially the same time each strum input signal is received. In ariff or note sequence mode, each strum and accompanying fret combinationmay be processed by the video game console to generate different notesequences. In modes involving note sequences, the note sequences areselected based on the fret combinations executed on the video gamecontroller. Each audio note sequence may begin at substantially the sametime as an executed strum, but may continue for a longer duration oftime, for example, several seconds, or for example, one minute. Asubsequent strum and different fret combination may lead the processorto stop outputting an old note sequence and initiate a new notesequence. In some embodiments, users are provided an option of selectingwhether to auto-loop a selected note sequence or riff and/or how manyiterations to repeat the note sequence or riff. In some embodiments, anote sequence mode may also include more intricate solo-style notesequences, or licks, which may also be initiated by the processor uponreceipt of a new set of strum and fret combination input signals. Insome embodiments the process may automatically loop some output sounds,for example riffs, unless the process receives a request or command, forexample in the form of a predetermined input signal or sequence ofpredetermined input signals, to cease looping of output sounds. In someembodiments, automatic looping of output sounds may cease, or change tonew output sounds, upon beginning of play or completion of a transitioncard, for example as discussed herein.

In embodiments including controller operation instructions orsuggestions, the instructions or suggestions may be presented to usersvisually in the form of, for example, different fret combinationsperformable on the video game controller. For example, gem-shaped iconsmay be displayed to represent different depressible fret buttons. Inembodiments of the invention with different modes, different shapedicons, or for example, icons with different hues or transparency levels,may be used to indicate to users the intended mode in which to performthe indicated fret combination instructions or suggestions with thevideo game controller.

In block 419, the process determines whether the tilt region has beenchanged. In some embodiments, tilt changes occur when the angularconfiguration of the guitar shaped video game controller is changed bythe user, either by pointing the neck of the video game controllerfurther up or further down with respect to a horizontal reference suchas the ground. The processor of the video game controller substantiallyconstantly processes input signals from the video game controllerindicating a current tilt position or status of the video gamecontroller to determine the tilt region the video game controllercurrently falls within. If a change in the tilt region is detected, theprocess returns to block 411, and a new active tilt region isdetermined. If no change in the tilt region is detected, the processcontinues to block 421. Some embodiments may allow users to lock intoand out of a particular tilt region by, for example, depression of apredefined button. Such a locking mechanism may be desirable to preventunintentional shifting into adjacent tilt regions. Tilting or moving theguitar while in the locked position in some embodiments allows for othermapping and uses for the tilt, motion and position inputs. Someembodiments may alternatively provide a form of auditory or visualfeedback to the user to notify the user that the controller has beenmoved into a different tilt region.

In block 421, the process determines whether to exit mode adjustment andgame processing. If the process determines to remain in mode adjustmentand game processing, the process returns to block 417, and the videogame continues being processed in the active mode. If, instead, theprocess determines to exit game processing, the process returns.

FIG. 5 is an illustration of a guitar shaped video game controller witha different mode assigned to each of three predefined tilt regions inaccordance with an embodiment of the invention. Guitar shaped video gamecontrollers such as the controller 501 illustrated in FIG. 5 may be usedin, for example, a musical video game as has been described herein. Inembodiments of the invention, the video game controller includes aposition or motion detection device, for example, an accelerometer,which generates information on a physical angular configuration of thevideo game controller with respect to a horizontal reference.

In typical embodiments, an accelerometer in the video game controllerdetects the tilt position of the video game controller and communicatesthe tilt position to a processor of an associated video game console.The tilt position may be, for example, an angle representative of theangle the axis of extension 511 of the video game controller neck 513makes with the ground. In these embodiments, if the neck of the videogame controller is parallel to the ground, the tilt position may becharacterized as 0 degrees. If the neck of the video game controller ispointed slightly upwards with respect to the ground, the angle may bedefined as a positive value. Likewise, if the neck is pointed downwardswith respect to the ground, the angle may be defined as a negativevalue.

In the embodiment of FIG. 5, the video game defines three tilt regionsin which the tilt position of the video game controller may reside. Eachof the three tilt regions includes information on upper and lowerangular limits. For example, a first tilt region 515 may have an upperlimit of 150 degrees above a horizontal reference and a lower limit of50 degrees above the horizontal reference, and a second tilt region 517may have, for example, an upper limit of 50 degrees above the horizontalreference, and a lower limit of 50 degrees below the horizontalreference. Furthermore, a specific mode of the video game may beassociated with each of the tilt regions. In FIG. 5, a riff mode isassociated with an upper tilt region, a regular note mode is associatedwith a central tilt region, and a power chord mode is associated with alower tilt region 519. As was discussed with respect to FIG. 4, each ofthe modes may be reserved or dedicated for outputting a particular typeor style of audio sample or samples, for example, the namesake riffs,notes, and power chords of each mode. Users may reposition 521 the videogame controller to be in different tilt regions to activate thedifferent modes of video game play. In other embodiments, more or lesstilt regions may be defined, and/or different mode or note mappings orassociations may be made to each defined tilt region.

In some embodiments, a background beat or rhythm may be establishedwhile the video game controller is positioned in one tilt region, andnotes or chords may be played over the beat or rhythm while the videogame controller is positioned in the same or a different tilt region.For example, some tilt regions may provide an option for repeating orlooping audio samples. In some embodiments, the looping samples areguitar-based, while in other embodiments, various other instruments, forexample, drums, or keyboard, may be featured in looping guide samples.Users may establish a repeating or looping audio track with selectedaudio samples, and subsequently play over the looping audio track withadditional notes or chords. In this fashion, a single user of the videogame system may generate multiple sources or layers of audio samplesthrough the video game console, and generate multi-dimensional audiocompilations on his or her own. Also, in some embodiments appropriatebackground beats, bass lines, guitar, vocals and other sounds areprovided in song sections based on the number of players, types ofsimulated instruments selected and the selected player levels.

FIG. 6 is a flow diagram of a process of presenting note suggestioncards to users in accordance with aspects of the invention. In someembodiments, multiple cards are simultaneously presented to a user, witheach card including information on suggested or recommended note orchord sequences for the user to follow through operation of a guitarshaped video game controller. The note suggestion cards may provideinformation on different series of fret combinations that may beexecuted on the video game controller. The input signals generated bythe fret combination sequences may be processed by the video gameconsole into audio output signals which fit harmoniously with, forexample, a background audio track being output simultaneously. In someembodiments, multiple different note suggestion cards are presented tothe user simultaneously, for example, three or four different cards asillustrated in the screen shot in FIG. 1. The cards present variousdifferent recommendations as to how to operate the video gamecontroller, and in most embodiments and instances are not obligatorycompliance instructions. More or less cards can be used in multiplayersituations. In addition for drums, continuous scrolling notes may beused instead of semi-static cards. In some embodiments, however,compliance with some note suggestion cards may serve as prerequisitesfor unlocking certain features of the video game, for example, new notesuggestions cards or new songs or background audio tracks. In someembodiments, information on note cards can scroll or highlight inpatterns that are in time with the song.

In block 611, the process determines the angular tilt region the guitarshaped video game controller is in. The configuration of the angulartilt regions may be similar to the angular regions described withrespect to block 315 of FIG. 3 and the illustration of FIG. 5. Inputsignals may be generated by the video game controller to communicate tothe processor of a video game console the particular tilt region inwhich the video game controller is currently positioned.

In block 613, the process updates the note mappings of the video gamecontroller. Each fret combination, and other available inputcombinations, may be mapped to different audio output samples orsignals, that is, the processor of the video game console processesinput signals indicating execution of different fret combinations togenerate various different audio output signals. In some embodiments,the fret combinations are mapped to different audio output or sampletypes depending on the tilt region in which the video game controller ispositioned. For example, in some embodiments, one tilt region isreserved for guitar power chord mappings, a second tilt region isreserved for single guitar note mappings, and a third tilt region isreserved for setting background audio track mappings. Therefore, aninput signal generated by a particular fret combination in one tiltregion may be processed by the video game console into a first audiooutput, while the same fret combination in a second tilt region may beprocessed into a second different audio output. The process thereforeupdates the note mappings to the various fret combinations based on thedetermined tilt region.

In block 615, the process selects multiple note suggestion cards basedon the particular tilt region the video game controller is currentlypositioned and the new note mappings. In embodiments where the tiltregion determines whether the video game controller outputs a note, achord, or a sequence of notes or chords, or even where tilt regionsaffect mappings of similar types of notes, particular note suggestionsequences may only be appropriate for select tilt regions. Therefore,the multiple note suggestion cards presented to the user may be tiltregion specific, and may be changed depending on the particular tiltregion the video game controller is in. The processor of the video gameconsole selects a new set of note suggestion cards appropriate for thecurrent tilt region and new note associations. In some embodiments,however, the selection of the multiple note suggestion cards may beperformed without reference to tilt of the video game controller.

In block 617, the process processes video game play. In embodiments withuser directed note generation, video game play may include a forum forusers to create audio output compilations, for example, musical songs oraudio tracks, through the video game console based on input signalsgenerated through the video game controller. The user may create his orher own musical compilations, or may opt to follow one or more of thenote suggestion cards presented by the video game console in the contextof video game play.

In block 619, the process determines whether the tilt region of thevideo game controller has been changed. During video game processing,the process may continually or periodically monitor the tilt level ofthe video game controller, and determine whether a tilt change hasoccurred. If a tilt change has not occurred, the process may proceed toblock 621, and eventually return to game processing in block 617. If achange to the tilt region has been detected, the process insteadproceeds back to block 611 to determine the new tilt region and make thenecessary adjustments to game play based on the new tilt region.

In block 621, the process determines whether to exit. If the processdetermines to remain in game play, the process returns to block 617, andcontinues processing video game play. If the process determines to exit,the process returns.

FIG. 7 is a screen shot of a user interface of a music based video gameduring user directed sound generation in accordance with aspects of theinvention. The user interface is the same interface presented andbriefly described with respect to FIG. 1. In some embodiments, the userinterface presented in FIG. 7 is used in conjunction with a music videogame including a guitar shaped video game controller, where the videogame console processes input signals from the video game controller togenerate output audio samples simulating real rhythm guitar, lead guitaror bass sounds. The user interface includes multiple note suggestioncards 701 running along the top of the screen which users may choose tofollow, a video game character 703 holding a guitar 705, which serves asa tilt level indicator and a tilt region indicator 707, and a scrollingmusic track 709 displaying the current fret fingering being executed onthe guitar shaped video game controller 711, as well as the mostrecently executed fret fingerings 713.

A music based video game with a user directed sound generation featureallows users to create their own musical compilations in a free playenvironment, without being obligated to comply with specific video gameinstructions. User directed sound generation closely emulates theplaying of a real instrument, through the use of a simulated instrumentsuch as the guitar shaped video game controller as has been describedherein.

Often in a free play environment, even with real instruments, musiciansmay desire some help or suggestions as to what to play. The userinterface therefore provides multiple note suggestion cards recommendingnote or chord sequences which may be executed on the video gamecontroller to generate pleasing audio outputs through the video gameconsole, which fit into the context of the background music or audiotrack being played. Each note suggestion card includes a series of fretcombinations to be depressed while operating a strum bar, with circularicons, for example, icons 715 indicating the fret buttons to bedepressed for each fret combination. Some embodiments also include anicon for an open note, signifying operation of the strum bar withoutdepression of any fret buttons, and some embodiments include icons forhammer ons and hammer offs, with the fret buttons operated without, attimes, operation of the strum bar. In some embodiments, as the playerplays the notes on a card, the completed notes and other actionindictors used to instruct the player to complete actuation combinationsin conjunction with their selected instrument are highlighted in realtime to display that they are completed. In some embodiments, a fullcard must be completed accurately to switch to a new card or otherstatic display. If a card is played incorrectly the completed notes arecleared and the player must start over. On the cards, each horizontalrow indicates a different fret combination, while vertical displacementseparates the suggested fret combinations sequentially. In theembodiment of FIG. 7, the sequences run from bottom to top, where usersmay perform a suggested note sequence presented on a card by executingeach fret combination starting with the bottommost fret combination andworking upwards. Typically, the cards are separated into five columns,each column corresponding to one of the fret buttons on a guitar shapedvideo game controller with five fret buttons. The circular icons indifferent columns may be colored or shaded differently, with each coloror shade matching the physical color of the fret buttons, or sliderbuttons, on the actual video game controller, for users to more easilyassociate the suggested fret combinations on screen with fret fingeringson the video game controller. Other indicators such as a wavy line caninstruct a user to hold a note, to move the whammy bar, or to slide afinger on the slider for example. Generally, the user interface providesthree or more note suggestion cards, sometimes including a transitioncard 717 at specific intervals of video game play, of which furtherdiscussion may be found with respect to FIG. 9 below.

The note suggestion cards may include various types of note sequences.For example, embodiments of the invention may show different notedurations, such as whole notes, quarter notes, eighth notes andsixteenth notes, etc. The note suggestion cards may also include otherinformation, for example, whether a fret fingering should be held aftera strum or whether the fret fingering should be staccato and abruptlyreleased, or for example, whether a hammer-on or pull-off should beperformed, that is, whether the fret fingering should be changed shortlyafter a strum. A similar visual system may be utilized in the suggestedscrolling display and presented in the user played scrolling display toreinforce learning of the distinct visual indicator types. In userdirected sound generation, users may operate the video game controllerin any way they desire, and there is generally no pressure to complywith the note suggestion cards. However, in some embodiments, successfulcompliance with a predefined amount of note suggestion cards, or withspecific note suggestion cards, may unlock additional cards or otherfeatures of the video game, providing a small incentive to comply withthe cards. In other embodiments, incentives may also include video gameanimations or effects involving a video game character or backgroundvenue.

At the bottom right of the user interface is a video game character,generally representing the user of the video game system. In someembodiments, the user may adjust the appearance of the video gamecharacter, for example, to more closely resemble the user, or forexample, to give the character a more distinct or more appealing look.In FIG. 7, the video game character is holding a guitar or a virtualinstrument in the shape of a guitar. The guitar serves as a tilt levelindicator, indicating the current tilt level at which the video gamecontroller is orientated. Along with the tilt level indicator isdisplayed tilt regions, including a currently active tilt region 719. Inthe embodiment of FIG. 7, there are three tilt regions, but otherembodiments of the invention may include more or less than three tiltregions depending on the configuration of the particular embodiment. Insome embodiments, the neck of the guitar protrudes into one of thedisplayed tilt regions, indicating that tilt region as the active tiltregion. In FIG. 7, the active tilt region is the center tilt region. Theactive tilt region may also be colored or shaded differently than theother tilt regions to identify it as the active tilt region

At the bottom left of the user interface is a scrolling music track. Thescrolling music track serves as an indicator of the fret fingeringcurrently being executed on the guitar shaped video game controller, aswell as recently fingered fret combinations. The scrolling track mayincrease and decrease speed based on the note density with respect totime. The scrolling music track provides a user with feedback as to thesequences of input signals the user has been generating, and how wellthe sequences comply with a particular note suggestion card they may betrying to follow. In some embodiments, the invention may be configuredto record into memory the fret combinations executed on the video gamecontroller and displayed on the scrolling music track, providing userswith access to their previously played compilations. Recorded or savedinformation may also include timing of the fret combinations, as well astilt information and information on successfully and unsuccessfullycomplied with cards during a recording session. In some embodiments, thescrolling music track may also provide the user with an enlarged orzoomed in view of a card the user has initiated compliance with, so thatthe user may be provided a larger and clearer view of the notesuggestions on a particular card.

FIG. 8 is a flow diagram of a process of replacing cards which have beencompleted during video game play in accordance with aspects of theinvention. Embodiments of the invention have been described where theuser interface simultaneously presents multiple note, pitch, beat, orother musical suggestion cards or semi-static displays to a user tosuggest different ways to operate, for example, fret buttons on a guitarshaped video game controller. If the process determines that one of themultiple note suggestion cards is being followed, the process determinesthe card being followed, and tracks the progress of the user and whetherhe ends up fully complying with the card.

In block 811, the process presents multiple note suggestion cards withdifferent note sequences. As was seen with respect to the notesuggestion cards presented in FIG. 7, there may be multiple differentnote sequences which the user may choose to follow and comply withthrough operation of the guitar shaped video game controller, or whichthe user may choose to ignore completely and freely operate the videogame controller.

In block 813, the process processes general game play. As has beenmentioned, in a user directed sound generation setting, the user maygenerate input signals using the video game controller. A processor of avideo game console receives the input signals and provides musical audiooutputs based on the particular input signals received. In this fashion,users may simulate the playing of a real instrument by using the videogame controller to create musical compilations through the video gameconsole.

In block 815, the process processes a first note of a note sequencespresented on one or more of the multiple note suggestion cards. The usermay execute a fret fingering, on the video game controller which matchesthe first fret combination of one or more of the note suggestion cards.The video game controller may generate input signals includinginformation on the fret combination, and the processor of the video gameconsole may generate a note it recognizes as a first note of one or moreof the suggested note sequences. In some embodiments, the processor thenhighlights the particular note suggestion card or cards, and awaitsfurther input signals from the video game controller to see if the userhas generated input signals compliant with the fret combinationspresented on any of the highlighted note suggestion cards. In someembodiments, the highlights can start over after a certain amount oftime or a mistake.

In block 817, the process determines compliance with a highlighted notesuggestion card. In some embodiments, the processor of the video gameconsole may track the subsequent input signals received from the videogame controller until one of two events occur. If the subsequent inputsignals generated by the user are not compliant with a suggested notesequence on a highlighted card at any point in the sequence, the processproceeds to block 821, where the process may begin awaiting anotherfirst note of at least one of the presented note sequences. If theprocess determines that the suggested note sequence of a highlightedcard is successfully complied with and completed, the process proceedsto block 819. In some instances, for example, instances where multiplecards have the same first note, there may be more than one highlightedcard. In such instances, the process may track compliance with each ofthe highlighted cards individually, and determine non-compliance only ifall of the highlighted cards are not complied with.

In block 819, the process replaces a successfully completed notesuggestion card with another card presenting a different suggested notesequence. In some embodiments, completion of note suggestion cards mayresult in the awarding of points or other bonuses in the context ofvideo game play. In other embodiments, completion of note suggestioncards reveals new suggestions, providing the user with new andinnovative note sequences to execute using the video game controller. Insome embodiments, note suggestion cards may also be arranged in minidecks or collections, where a note sequence starting on a first card iscontinued on a subsequent card, so that multiple subsequent cards mayfit together to form a longer suggested note sequence. In someembodiments, the processor may also include more intricate cardreplacement animations. For example, a note suggestion card may startburning or showing a similar highlight from bottom to top while the notesequence on the card is being complied with, to partially reveal theunderlying card or discoloring the played notes or beats

In block 821, the process determines whether to exit the card complianceand replacement process of FIG. 8. If the process determines to remainin card compliance and replacement, the process returns to block 813 andcontinues processing the game. If the process determines to exit, theprocess returns.

FIG. 9 is a flow diagram of a process of transitioning from a firstsection of a song to a second section of a song in accordance withaspects of the invention. The process may be performed for example by aprocessor or a video game console. Often, songs may be separated intodifferent sections, for example, an introduction, a chorus, a solosection, and a bridge section. In addition, in some instances atransition may be provided between song sections, and at times thetransition may also be considered a song section. Further, particularnote sequences may be used for select song sections and not for others.For example, an introduction typically includes a serene rhythm or asofter section of the song, in many instances to build anticipation forthe song. Other examples include note sequences in choruses or bridgeswhich provide catchy rhythms, or solo sections which provide moreintricate and difficult note sequences. Some embodiments of theinvention, therefore, separate note suggestion cards into differentbatches, with cards drawn and presented from, and in some embodimentsonly from, batches which are appropriate for a current song section, andthe current song section may change during video game play, generally aspart of video game play.

In block 911, the process provides cards from a first card batch inaccordance with a current song section. The process determines thecurrent song section, and determines the specific batches of cards fromwhich to draw note suggestions cards to present in the user interface.For example, at the outset of user directed sound generation, the songsection may be set as an introductory section, with cards selected froma batch with note sequences appropriate for a song introduction. In themiddle of a song, after one or more transitions for example, the songmay be in a rhythmic or bridge section, and the process may select cardsfrom a batch reserved for rhythmic or bridge sections of the song.

In block 913, the process determines that prerequisites for a transitionto a new section have been met. In some embodiments of the invention,the process provides for transition opportunities into new sections of asong when one or a set of prerequisites are met by the user. Examples ofprerequisites may include, for example, successful compliance with apredefined number of note suggestion cards, successful compliance with apredetermined note suggestion, or receipt of particular input signalsindicative of a request or command to transition to a different songsection. In still other embodiments the prerequisites may be passage ofa certain duration of time in a song section, for example 30 seconds orone minute. In other embodiments, prerequisites for transitioning maynot be established, and the user may be allowed to freely transitionfrom one song section to a different song section through play of aparticular note sequence of a card presented on the display providing atransition, or by some other action.

In block 915, the process provides a transition card to the user. Insome embodiments, a transition card is a card with a note or notesequence suggestion, which upon completion, transitions the video gamefrom one song section into a different song section. The transition cardmay be similar to or the same as a normal note suggestion card, andinclude a particular note sequence, which may be, for example, a popularand easily recognizable note sequence commonly used to bridge twosections of songs together, although in most embodiments the particularnote sequence, and all note sequences presented on the cards, is anoriginal note sequence created specifically for use in a video game. Insome embodiments, the transition card may be easier to comply with, forexample, the card instructing play of a single chord with a three orfour fret button combination. The transition card will generally bepositioned in the user interface apart from other note suggestion cardsto differentiate it from other note suggestion cards. For example, inthe user interface of FIG. 7, the transition card may be the rightmostcard, spaced apart and distinguishable from the other note suggestioncards. In other embodiments the transition card may be distinguishableby color, or by shape, or by some combination of color and shape. Insome embodiments, the transition card appears regardless of the activetilt region. In other embodiments, the transition card may only appearwhen the video game controller is positioned in certain tilt regions,for example, to prevent a user from accidentally transitioning into adifferent section of the song while in the other tilt regions.

In block 917, the process determines user compliance with the transitioncard. Compliance with the transition card is similar to compliance withregular note suggestion cards. A fret combination or series of fretcombinations is presented to the user, and the user attempts to followor match the fret combination or combinations by operating the fretbuttons and/or strum bar of a guitar shaped video game controller.Successful compliance with a transition card results in a transitionfrom a first song section to a second different song section. Forexample, if video game play is currently in a chorus song section, and atransition card appears on the user interface, the user may choose tosuccessfully comply with the transition card. Upon compliance with thetransition card, the video game may transition the song into asubsequent section, for example, a solo section of the song.

In block 919, the process provides cards from a second card batchassociated with a further song section. In most embodiments, uponcompletion of a transition card, the process may replace all the notesuggestion cards displayed on the user interface with a new set of notesuggestion cards. The prior note suggestion cards may be from a batchassociated with a different song section, and may not be appropriate forthe new song section. The replacement note suggestion cards arepreferably drawn from a different batch of cards associated with the newsong section to ensure that the newly presented note sequencesharmoniously fit into the context of the new song section. After theprocess provides note suggestion cards from the new card batch, theprocess returns. In some embodiments the combination of cards and noteson cards of the new card batch may be generated by the process accordingto preset rules, for example rules varying card information if a usertransitions to the same section repeatedly.

FIG. 10 is a flow diagram of an alternate process of determining audiooutputs in accordance with aspects of the invention. A video gamecontroller, for example, the video game controller described withrespect to FIG. 1, is operated by a user and generates input signalsincluding information on the user operation of different input devicesof the video game controller. Such an alternate embodiment may only beactive in certain tilt regions, for example, the tilt regioncorresponding to a riff mode, or may be activated by a predefined inputor sequence of inputs. In a music video game similar to the onedescribed with respect to FIG. 1, audio outputs may be in the form ofaudio samples recorded from instruments, for example, guitars, drums, orkeyboards. In some embodiments, the audio samples may be MIDI basedaudio samples. Some embodiments may provide users the option ofestablishing a background audio track using audio samples of a firstinstrument or set of instruments, for example, a drum set, andthereafter perform single notes, chords, or note sequences with audiosamples of a second instrument or set of instruments, for example, anelectric guitar or a keyboard, over the pre-established background audiotrack.

In block 1011, the process receives a fret input or inputs from thevideo game controller. In some embodiments, the video game controllergenerates an input signal when a fret input or multiple fret inputs aredepressed. In other embodiments, the video game controller maycontinually generate input signals including information on status ofthe fret inputs, and a processor of a video game console may detect achange in the status when, for example, one or more of the fret input isdepressed or released. In yet other embodiments, an input signal may begenerated by the video game controller when a strum input is deflectedupwards or downwards, with information on the status of each fret inputdepressed at the time the strum input was deflected.

In block 1013, the process selects sets of MIDI sequences to output. Inmost embodiments of the invention, each fret input or fret inputcombination is mapped to a different set of MIDI sequences. Someembodiments may feature different drum performances including MIDI-basedaudio tracks from multiple different types of drums, including multipleaudio samples of different drum components of a drum set. Such aperformance may include, for example, separate snare drum, kick drum,and hi-hat audio tracks, as well as additional audio tracks featuringfurther drum components. Drums, drum fills and drum beats can be playedwith cards on a guitar controller. On beginner levels basic beats can beprovided to keep the band in time. Many of these performances may beprerecorded and stored in, for example, the video game CD-ROM, or mayhave been previously created or recorded by a user in a different modeor feature of the particular video game. In these embodiments, a firstfret input may be mapped to a collection of MIDI-based audio tracksmaking up a first drum performance, while a second fret input may bemapped to a collection of audio tracks making up a second, different,drum performance, using the same or similar drum set. Therefore, eachfret input may be mapped to different drum performances, with eachperformance including audio tracks using the same drum components, forexample, snare, kick, hi-hat, or crash cymbal. In some embodiments,video game information found on, for example, a video game CD-ROM, mayinclude a lookup table providing for different sets of MIDI sequencemappings to various fret inputs or fret input combinations. In someembodiments, audio samples may include MIDI samples from otherinstruments, or for example, non-MIDI audio samples, depending on eachindividual embodiment.

In block 1015, the process receives an input signal from an analog touchpad on the video game controller. In embodiments using the guitar shapedvideo game controller described with respect to FIG. 1, the input signalmay be generated by the video game controller when, for example, afinger or other actuator depresses a portion of the analog touch pad,with the input signal including information on the portion of the touchpad depressed. In some embodiments, input signals are continuallygenerated by the video game controller and sent to a processor of thevideo game controller, and may include status information on the statusof, for example, the analog touch pad. In these embodiments, theprocessor may respond to any status changes detected by the touch pad.Alternatively, the process may receive a variety of different inputsignals, for example, an input signal indicating operation of any inputdevice available on the video game controller other than the fretinputs, or any secondary input signal type different than a primaryinput signal type used by the process in block 1013 selecting initialMIDI sequences. Depending on the configuration of the video gamecontroller used in conjunction with the invention, the process mayspecify or designate different primary and secondary input sources.

In block 1017, the process applies MIDI filters, or similar filters forfiltering different parts of the selected MIDI performances orcompilations, for example, in block 1013 of the process. In someembodiments, the process may apply the filters based on, for example,the input signals generated by the analog touch pad in block 1015.Different filters may be associated with different input signals fromthe analog touch pad, much like different MIDI sequences were mapped todifferent fret inputs in block 1013. For example, for a drum setsequence, depression of a first portion of the analog touch pad mayfilter out a snare drum MIDI sequence, depression of a second portion ofthe analog touch pad may filter out a kick drum MIDI sequence, anddepression of a third portion of the analog touch pad may filter out allMIDI sequences besides the kick drum sequence, or alternatively addthose elements. In embodiments using the General MIDI protocol, audiosamples for each drum component are assigned a different channel andnote designator. For example, the snare drum sample is assigned channel10, note 38 in the General MIDI protocol, while the hi-hat sample isassigned channel 10, note 46. In these embodiments, applying a filtermay include blocking a particular MIDI audio sample from being output byblocking specific channel/note combinations in the MIDI protocol. Inother embodiments, the process may not use the MIDI protocol, in whichcase audio sample filtering may be accomplished in a variety ofalternate ways.

In block 1019, the process outputs the unfiltered MIDI sequences. A usermay thus operate the video game controller to select a desiredprerecorded drum cadence or sequence by operation of a first set ofinput devices, for example, the fret inputs on the guitar shaped videogame controller as has been described herein. The user may then operatethe video game controller to filter out different components of the drumsequence by operation of a second input device or set of input devices,for example, the analog touch pad, tilt, or whammy. In this fashion, auser may generate and modify a wide range of audio samples throughsimple manipulation of the video game controller. Depending on theembodiment, audio samples of other instruments may also be output inconjunction with the invention, for example, keyboard audio samples orguitar audio samples. After the unfiltered MIDI sequences are output,the process returns.

FIG. 11 is a flow diagram of a dynamic process of determining audiooutputs in accordance with aspects of the invention. FIG. 11 may beemployed during game play of, for example, a music based video game, togenerate different audio samples from a video game console throughoperation of an associated video game controller. In many respects, theprocess of FIG. 11 is similar to the process of FIG. 10, with theaddition of real-time adjustments to either the MIDI sample selectionsor the applied filters through user manipulations of the video gamecontroller. While the following discussion focuses on a preferredembodiment using specific input devices on a guitar shaped video gamecontroller, it is to be understood that different input devices anddifferent audio sample formats may be employed depending on the detailsof each individual embodiment, many variations of which were alluded towith respect to FIG. 10.

In block 1111, the process receives a fret input signal. A fret inputsignal may be generated by the video game controller through specificoperation of a plurality of available fret inputs. The video gamecontroller may generate an input signal including information ofdepression of one or more fret inputs, and send the input signal to aprocessor of a video game console. Alternatively, the processor of thevideo game console may monitor the status of all the fret inputs anddetect changes in the status.

In block 1113, the process determines a set of MIDI samples mapped to aparticular fret input signal. Most embodiments select MIDI sample setsbased on the fret input signal received in block 1111. The MIDI samplesets may be drum-based, including individual parts for, for example, asnare drum, a kick drum, and a hi-hat cymbal. A collection of MIDIsamples generally combine to create a musical performance, such as adrum solo or fill, and generally have a duration lasting between a fewbeats and multiple measures of music. In most embodiments, once a set ofMIDI samples concludes, the set is looped back to the beginning andrepeated. The MIDI samples may alternatively be, for example,guitar-based, and include individual parts for a lead guitar, a rhythmguitar, and a bass guitar.

In block 1115, the process plays the selected set of MIDI samples. Inmany instances, a selected set of MIDI samples may be used by a noviceuser of the video game to play more advanced sequences of notes withminimal practice. In other instances, a selected set of MIDI samples maybe utilized in the context of a music video game as a source ofbackground audio or background beat. Users in these instances may beallowed to command the video game console to output one or more of thesets of background MIDI samples, and subsequently play individual notessimultaneously with the background samples.

In block 1117, the process determines whether to filter one or moreindividual MIDI samples or tracks from a selected set or collection ofMIDI samples being output. Generally, multiple MIDI samples are beingoutput by the process at any given time. Through user operation ofpredefined input devices on the video game controller, for example, theanalog touch pad located on the neck of the guitar shaped video gamecontroller as described in FIG. 1, a user may dictate which of the MIDIsamples being output by the video game console are to be muted orfiltered out. If an input signal corresponding to application of a MIDIfilter is detected, the process proceeds to block 1119. If, however, aninput signal corresponding to application of a MIDI filter is notdetected, the process instead proceeds to block 1121.

In block 1119, the process filters individual MIDI samples from a set ofMIDI samples being output by the video game console. The specific MIDIsamples filtered or blocked may be determined based on an input signalor signals received from, for example, the analog touch pad, as wasdescribed with respect to block 1117. A performance comprised ofseparate MIDI samples may include sounds generable from a plurality ofdifferent instruments. For example, in a drum performance, differentcomponents of a drum set may be represented in any given performance.Under the General MIDI protocol, each individual instrument type may bedefined by a channel and note, and be identified as such by theprocessor of the video game console. For example, different drum samplesunder the General MIDI protocol may be identified by differentchannel/note combinations as was described above with respect to block1017 of the process of FIG. 10. Different filtering mechanisms maydesignate particular MIDI channel/note combinations, corresponding todifferent MIDI samples, to filter. For example, one particular inputsignal from the analog touch pad may be identified by the processor forfiltering or blocking channel 10, note 46, corresponding to audiosamples associated with a hi-hat cymbal, from being output by the videogame console. Other input signals generated through the analog touch padmay filter other MIDI samples from a set of audio samples in a similarfashion. After filtering is complete, the processor returns to block1115 and outputs the remaining unfiltered MIDI samples.

In block 1121, the process determines whether a fret change has beenreceived or detected. In most embodiments, each fret input or fret inputcombination is mapped to a prerecorded set of MIDI samples, for example,a performance utilizing multiple instruments, or alternatively, multiplecomponents of a dynamic instrument such as a drum set. User depressionof a different fret input or fret input combination may be processed bythe video game console to output a different prerecorded set of MIDIsamples. In this fashion, a single fret input depression may lead togeneration of a musical composition including multiple instrumentalparts through the video game console. Changing of the fret input orinputs may thereafter lead to generation of a different composition, sothat even more novice users may be able to easily generate intricatepieces of music in the context of the music video game. If the processdetermines a fret change has occurred, the process returns to block 1111to determine the new fret input signal. If the process determines thatno fret change has occurred, the process proceeds to block 1123.

In block 1123, the process determines whether to exit audio outputting.If the process determines to continue outputting audio samples in thecontext of the music video game, the process returns to block 1115 andcontinues to output MIDI samples. If, instead, the process determines toexit audio outputting, the process returns.

FIG. 12 is a flow diagram of a process of generating and recording amulti-dimensional or polyphonic audio compilation in accordance withaspects of the invention. In some embodiments, a riff mode may beavailable. For example, in some embodiments a user may set and loop aselected riff or rhythmic sequence of notes as a background beat oraudio source, while in some embodiments the process may determine a riffor riffs to play. In these embodiments, users may subsequently generatenotes over the established background audio, and in some embodimentsusers may select alternative riffs as background audio or operate avideo game controller so as to result in the process receiving inputsignals from the video game controller instructing the process to ceaseplaying some or all of the background audio. In some embodiments,repeating and looping may not be limited to a riff mode, but may insteadbe available for all modes, whether the mode is typically used forgeneration of note sequences or single notes or chords. Some embodimentsmay further provide non-guitar based audio loops, for example, drumbackground beats, for use as repeating background audio.

In block 1211, the process selects a background audio beat or riff. Thebackground audio beat or riff may be based on an input signal or set ofinput signals generated and sent by the video game controller.Generally, background audio samples are selected based on a set of strumand fret combination input signals received from the video gamecontroller. In embodiments where riffs are primarily used as backgroundaudio tracks, background audio selection may occur while the video gamecontroller is positioned in a tilt region associated with a riff mode.In some embodiments, the user may also be provided options for adjustingrepeating or looping properties of the selected background audio. Forexample, particular input signals from the video game controller may beprocessed by the video game console, for example, for adjusting thenumber of times the background audio is repeated, or for example, foradjusting the volume of the established background audio. In someembodiments, methods of disabling a looping background audio sample mayalso be provided. Disabling an auto-looping background audio sample mayinvolve, for example, the depression of predefined input devices on thevideo game controller, or for example, the initiation of a replacementbackground audio sample.

In block 1213, the process generates a note or chord over the selectedbackground audio. In some embodiments, the notes or chords simulate realguitar sounds. The processor of the video game console selects andoutputs audio notes and chords based on input signals received from thevideo game controller. In some embodiments, the video game controller ismoved into a tilt region associated with a note mode or a power chordmode to generate notes over the background audio sample. In general, thebackground audio sample continues running while single notes or chordsare output by the video game console.

In block 1215, the process determines whether to record the audiooutputs. If the process determines to record or save the audio outputs,the process proceeds to block 1217. If instead the process determinesnot to record the audio outputs, the process proceeds to block 1219.

In block 1217, the process saves the input signals associated with thebackground audio sample, as well as the input signals associated withthe generated note or chord played over the background audio sample.Both the background audio sample and a generated note or chord areestablished using input signals from the video game controller.Recording of the entire audio compilation therefore involves saving theinformation involved in establishing both the background audio and thegenerated note or chord. In typical embodiments, the information mayinclude the fret combination mapped to the background audio sample, aswell as the fret combination mapped to the note or chord output by thevideo game console. The process may thus be repeated to create a musicalcompilation including two or more audio tracks generated by a singleuser of the invention. In some embodiments the saved information may beuploaded to a server of a network, for example the Internet, anddownloaded to other users for video game play or other enjoyment.

In block 1219, the process determines whether to exit music generationand recording. If the process determines to remain in music generationand recording, the process returns to block 1213, and generates afurther note over the established background audio sample. If, instead,the process determines to exit music generation and recording, theprocess returns.

The invention therefore provides for a user directed sound generationfeature in a music video game, with provided sound generationsuggestions or recommendations. Although the invention has beendescribed with respect to certain embodiments, it should be recognizedthat the invention may be practiced other than as specificallydescribed, the invention comprising the claims and their insubstantialvariations supported by this disclosure.

What is claimed is:
 1. A method of complying with one of a plurality of suggested note sequences in a musical video game, comprising: presenting, simultaneously, a plurality of sequences of suggested user inputs on a display, each of the sequences of suggested user inputs providing different recommendations as to operation of a video game controller; receiving an input signal generated based on user operation of input devices on the video game controller, the input signal in compliance with a first suggested input of a one of the plurality of sequences of suggested user inputs, the one of the plurality of sequences of suggested user inputs being any one of the plurality of sequences of suggested user inputs; determining compliance of subsequently received input signals from the video game controller with the one of the plurality of sequences of suggested user inputs; and replacing, on the display, the one of the plurality of sequences of suggested user inputs with a new sequence of suggested user inputs.
 2. The method of claim 1, wherein each sequence of the plurality of sequences of suggested map audio samples complementary to a background audio track.
 3. The method of claim 2, wherein the plurality of sequences of suggested user inputs are replaced by a further plurality of sequences of suggested user inputs if the provided background audio track is modified.
 4. The method of claim 1, wherein the video game controller is a guitar shaped video game controller, and the input devices include a plurality of fret inputs, and wherein the sequences of suggested user inputs include information of operation of the plurality of fret inputs and timing of the operation of the plurality of fret inputs.
 5. The method of claim 4, wherein the video game controller further includes a touch pad input, and wherein the sequences of suggested user inputs include status information of the touch pad input.
 6. The method of claim 1, wherein three sequences of suggested user inputs are presented.
 7. The method of claim 1, further comprising determining that the subsequently received input signals form the video game controller complied with the one of the plurality of sequences of suggested user inputs, and wherein the replacing the one of the plurality of sequences of suggested user inputs with the new sequence of suggested user inputs is in response to determining that the subsequently received input signals from the video game controller complied with the one of the plurality of sequences of suggested user inputs.
 8. The method of claim 1, further comprising highlighting presentation of the one of the plurality of sequences of suggested user inputs for which the input signal in compliance with the first suggested input was received. 